Constraining neural networks for robustness through alternative encoding

ABSTRACT

A neural network is augmented to enhance robustness against adversarial attack. In this approach, a fully-connected additional layer is associated with a last layer of the neural network. The additional layer has a lower dimensionality than at least one or more intermediate layers. After sizing the additional layer appropriately, a vector bit encoding is applied. The encoding comprises an encoding vector for each output class. Preferably, the encoding is an n-hot encoding, wherein n represents a hyperparameter. The resulting neural network is then trained to encourage the network to associated features with each of the hot positions. In this manner, the network learns a reduced feature set representing those features that contain a high amount of information with respect to each output class, and/or to learn constraints between those features and the output classes. The trained neural network is used to perform a classification that is robust against adversarial examples.

BACKGROUND Technical Field

This disclosure relates generally to information security and, inparticular, to protect a neural network against adversarial attack.

Background of the Related Art

Machine learning technologies, which are key components ofstate-of-the-art Artificial Intelligence (AI) services, have shown greatsuccess in providing human-level capabilities for a variety of tasks,such as image recognition, speech recognition, and natural languageprocessing, and others. Most major technology companies are buildingtheir AI products and services with deep learning models (e.g., deepneural networks (DNNs)) as the key components. Building aproduction-level deep learning model is a non-trivial task, whichrequires a large amount of training data, powerful computing resources,and human expertise. For example, the creation of a Convolutional NeuralNetwork (CNN) designed for image classification may take from severaldays to several weeks on multiple GPUs with an image dataset havingmillions of images. In addition, designing a deep learning modelrequires significant machine learning expertise and numeroustrial-and-error iterations for defining model architectures andselecting model hyper-parameters.

Deep learning has been shown to be effective in a variety of real-worldapplications such as computer vision, natural language processing andspeech recognition. It also has shown great potentials in clinicalinformatics such as medical diagnosis and regulatory decisions,including learning representations of patient records, supportingdisease phenotyping and conducting predictions. Recent studies, however,show that these models are vulnerable to adversarial attacks, whichattacks are design to intentionally inject small perturbations (alsoknown as “adversarial examples”) to a model's data input to causemisclassifications. In image classification, researchers havedemonstrated that imperceptible changes in input can mislead theclassifier. In the text domain, synonym substitution or character/wordlevel modification on a few words can also cause the model tomisclassify. These perturbations are mostly imperceptible to humans butcan easily fool a high-performance deep learning model.

Since the discovery of adversarial examples, numerous techniques havebeen proposed to defend neural networks against them, but suchtechniques either are expensive, or brittle. To date, only dataaugmentation has been shown to be effective. The first data augmentationdefense, adversarial training, involves iteratively generatingadversarial examples during training and adding these samples asadditional training inputs. Although integrating adversarial samples inmodel training is effective in defending neural networks against theattack used to generate the adversarial training examples, it iscomputationally-expensive due to the generation process, and it is onlyeffective against the specific type of adversarial examples used duringtraining.

Another augmentation defense of note is randomized smoothing, which isan approach that augments the training dataset with a noisy version ofthe dataset crafted using Gaussian noise. This approach is premised onthe notion that adversarially-robust models also tend to be robustagainst naturally noisy inputs. Although randomized smoothing reducesthe computational overhead introduced by adversarial training (as noadversarial samples need to be generated), the approach isdisadvantageous as it weakens the final model comparatively.

In addition to these data augmentation defenses, there have beennumerous other methods proposed to improve the adversarial robustness ofneural networks, e.g., through the use of pre-processing techniques, ornew types of network layers. Many of these techniques rely on gradientshattering, or some variant thereof. In particular, traditional whitebox adversarial attacks use a network's loss gradient to craftadversarial samples. Gradient shattering breaks these attacks by causingthe loss gradient to vanish, usually due to a non-differentiableoperation. Prior work has shown that gradient shatter is an ineffectivedefense, as an adversary is able to adapt, e.g., by skipping over thenon-differentiable operations, or by using a rough approximation of thedifferentiable operation.

Thus, there remains a need in the art to provide techniques to ensurethe reliability and robustness of neural network classifiers in the faceof adversarial examples.

BRIEF SUMMARY

The technique herein provides for creating and operating anadversarially-robust neural network. The neural network that isconfigured for adversarial robustness typically comprises a first orinput layer, a last or output layer, and one or more intermediate layersthat may be hidden. In this approach, the network is augmented toinclude one or more additional layers that are configured with anencoding scheme that provides an alternative to a traditional one-hotencoding for classes. Preferably, the alternative encoding is avector-based encoding wherein a particular class label is represented bya vector of 0s and 1s. In one example embodiment, at least oneadditional layer of this type is positioned between an intermediatelayer and the last layer, and wherein the additional layer has a lowerdimensionality (i.e., a lesser number of neurons) than the one or moreintermediate layers. The alternative layer preferably is afully-connected layer, and it is sized to include a number of neuronssufficient to uniquely label an output class of the network (using theencoding). Thus, for example, if the neural network is a classifier thanhas ten (10) output classes, the alternative layer is sized to includeat least four (4) neurons, which in this example represents four bitpositions (≥√{square root over (10)}) representing an encoding of theten output classes. After including the additional layer and determiningits size (given the number of output classes), an encoding is applied tothe weights in the layer. The weights comprise a weight matrix, witheach row of the matrix being an i^(th) encoding vector for the i^(th)output class. This alternative encoding is sometimes referred to hereinas an “n-hot” encoding to distinguish it from traditional 1-hotencoding, and wherein a value of “n” is configured as a hyperparameter.The neural network classifier as augmented to include this additionallayer is then trained, typically in a conventional manner. Based on thetraining, the encoding causes the network to associate a reduced numberof active features with each of the hot positions (namely, the 1s in theweight matrix), thereby encouraging the network to learn those featuresthat contain a high amount of information with respect to each class.Thus, the additional layer limits (constrains) the network and, inparticular, on the number of features used to predict each class, and/orthe layer adds constraints between those features and the outputclasses. Once trained in this manner, the adversarially-robust neuralnetwork is then applied to a classification task.

The foregoing has outlined some of the more pertinent features of thesubject matter. These features should be construed to be merelyillustrative. Many other beneficial results can be attained by applyingthe disclosed subject matter in a different manner or by modifying thesubject matter as will be described.

BRIEF DESCRIPTION OF THE DRAWINGS

For a more complete understanding of the subject matter and theadvantages thereof, reference is now made to the following descriptionstaken in conjunction with the accompanying drawings, in which:

FIG. 1 depicts a representative deep learning model (DNN);

FIG. 2 depicts a deep learning model used in association with a deployedsystem or application;

FIG. 3 depicts augmentation of the neural network in FIG. 1 to includean n-hot encoding layer according to this disclosure;

FIG. 4 is a representative process flow for constructing and applying ann-hot encoding used in an additional layer of the neural networkaccording to one embodiment; and

FIG. 5 is a block diagram of a data processing system in which exemplaryaspects of the illustrative embodiments may be implemented.

DETAILED DESCRIPTION OF AN ILLUSTRATIVE EMBODIMENT

As will be seen, the technique herein provides for enhancing therobustness of a neural network against adversarial attack. By way ofbackground, the following provides basic principles of deep learning.

As is well-known, deep learning is a type of machine learning frameworkthat automatically learns hierarchical data representation from trainingdata without the need to handcraft feature representation. Deep learningmethods are based on learning architectures called deep neural networks(DNNs), which are composed of many basic neural network units such aslinear perceptrons, convolutions and non-linear activation functions.Theses network units are organized as layers (from a few to more than athousand), and they are trained directly from the raw data to recognizecomplicated concepts. Lower network layers often correspond withlow-level features (e.g., in image recognition, such as corners andedges of images), while the higher layers typically correspond withhigh-level, semantically-meaningful features.

Specifically, a deep neural network (DNN) takes as input the rawtraining data representation and maps it to an output via a parametricfunction. The parametric function is defined by both the networkarchitecture and the collective parameters of all the neural networkunits used in the network architecture. Each network unit receives aninput vector from its connected neurons and outputs a value that will bepassed to the following layers. For example, a linear unit outputs thedot product between its weight parameters and the output values of itsconnected neurons from the previous layers. To increase the capacity ofDNNs in modeling the complex structure in training data, different typesof network units have been developed and used in combination of linearactivations, such as non-linear activation units (hyperbolic tangent,sigmoid, Rectified Linear Unit, etc.), max pooling and batchnormalization. If the purpose of the neural network is to classify datainto a finite set of classes, the activation function in the outputlayer typically is a softmax function, which can be viewed as thepredicted class distribution of a set of classes.

Prior to training the network weights for a DNN, an initial step is todetermine the architecture for the model, and this often requiresnon-trivial domain expertise and engineering efforts. Given the networkarchitecture, the network behavior is determined by values of thenetwork parameters. More formally, let D={x_(i), z_(i)}^(T) _(i=1) bethe training data, where z_(i)·[0, n−1] is a ground truth label forx_(i), the network parameters are optimized to minimize a differencebetween the predicted class labels and the ground truth labels based ona loss function. Currently, the most widely-used approach for trainingDNNs is a back-propagation algorithm, where the network parameters areupdated by propagating a gradient of prediction loss from the outputlayer through the entire network. Most commonly-used DNNs arefeed-forward neural networks, wherein connections between the neurons donot form loops; other types of DNNs include recurrent neural networks,such as long short-term memory (LSTM), and these types of networks areeffective in modeling sequential data.

Formally, a DNN has been described in literature by a function g: X→Y,where X is an input space, and Y is an output space representing acategorical set. For a sample x that is an element of X,g(x)=f_(L)(F_(L-1)( . . . ((f_(l)(x)))). Each f_(i) represents a layer,and F_(L) is the last output layer. The last output layer creates amapping from a hidden space to the output space (class labels) through asoftmax function that outputs a vector of real numbers in the range [0,1] that add up to 1. The output of the softmax function is a probabilitydistribution of input x over C different possible output classes.

FIG. 1 depicts a representative DNN 100, sometimes referred to anartificial neural network. As depicted, DNN 100 is an interconnectedgroup of nodes (neurons), with each node 103 representing an artificialneuron, and a line 105 representing a connection from the output of oneartificial neuron to the input of another. In the DNN, the output ofeach neuron is computed by some non-linear function of the sum of itsinputs. The connections between neurons are known as edges. Neurons andthe edges typically have a weight that adjusts as learning proceeds. Theweight increases or decreases the strength of the signal at aconnection. As depicted, in a DNN 100 typically the neurons areaggregated in layers, and different layers may perform differenttransformations on their inputs. As depicted, signals (typically realnumbers) travel from the first layer (the input layer) 102 to the lastlayer (the output layer) 104, via traversing one or more intermediate(the hidden layers) 106. Hidden layers 106 provide the ability toextract features from the input layer 102. As depicted in FIG. 1 , thereare two hidden layers, but this is not a limitation. Typically, thenumber of hidden layers (and the number of neurons in each layer) is afunction of the problem that is being addressed by the network. Anetwork that includes too many neurons in a hidden layer may overfit andthus memorize input patterns, thereby limiting the network's ability togeneralize. On the other hand, if there are too few neurons in thehidden layer(s), the network is unable to represent the input-spacefeatures, which also limits the ability of the network to generalize. Ingeneral, the smaller the network (fewer neurons and weights), the betterthe network.

The DNN 100 is trained using a training data set, thereby resulting ingeneration of a set of weights corresponding to the trained DNN.Formally, a training set contains N labeled inputs where the i^(th)input is denoted (x_(i), y_(i)). During training, parameters related toeach layer are randomly initialized, and input samples (x_(i), y_(i))are fed through the network. The output of the network is a predictiong(x_(i)) associated with the i^(th) sample. To train the DNN, thedifference between a predicted output g(x_(i)) and its true label,y_(i), is modeled with a loss function, J (g(x_(i)), y_(i)), which isback-propagated into the network to update the model parameters.

Typically, a neural network model such as depicted in FIG. 1 acceptsnumeric values as inputs. To work with categorical data, however, suchdata typically needs to be encoded in some manner. One-hot encoding is aknown technique that converts category data into integers or a vector ofones and zeros. In this approach, the length of the vector is dependenton the number of expected classes or categories, and each element in thevector represents a class. In one-hot encoding, a one is used toindicate the class, and all other values in the vector are zero. Statedanother way, if category data is not ordinal, one-hot encoding thusprovides a useful way to work with such data. Label encoding convertscategorical variables to numerical representations, which aremachine-readable.

FIG. 2 depicts a DNN 200 deployed as a front-end to a deployed system,application, task or the like 202. The deployed system may be of anytype in which machine learning is used to support decision-making. Asnoted above, neural networks such as described are vulnerable toadversarial attacks, which attacks are design to intentionally injectsmall perturbations (“adversarial examples”) to a model's data input tocause misclassifications. In image classification, researchers havedemonstrated that imperceptible changes in input can mislead theclassifier. In the text domain, synonym substitution or character/wordlevel modification on a few words can also cause the model tomisclassify. These perturbations are mostly imperceptible to humans butcan easily fool a high-performance deep learning model. For exemplarypurposes, it is assumed that the above-described DNN 100 (FIG. 1 ) orDNN 200 (FIG. 2 ) is subject to adversarial attack. The technique ofthis disclosure is then used to enhance the robustness of that network.The resulting network is then said to be adversarially-robust, meaningthat—as compared to the network that does not incorporate the describedtechnique—the resulting network is better able to provide the requisiteclassification task even in the face of adversarial examples.

The particular neural network, the nature of its classification, and/orthe particular deployment system or strategy are not limitations of thetechnique herein, which may be employed to strengthen any type ofnetwork classifier regardless of its structure and use.

With the above as background, the technique of this disclosure is nowdescribed.

Constraining Neural Networks for Robustness Through Alternative Encoding

With reference to FIG. 3 , the standard DNN 300, as shown on the left,is converted to an adversarially-robust neural network 302, as shown onthe right. In this example, the standard DNN 300 comprise an input layer304, hidden (intermediate) layers (L1, L2 and L3) 306, and output(penultimate) layer 305. The adversarially-robust neural network 302likewise comprises input layer 310, output layer 312, and hidden layers314. Standard training creates network 300, but that network is composedof both robust and non-robust features. As such, the network 300 is notadversarially-robust. To address this deficiency, the technique of thisdisclosure augments the DNN 300 to create the adversarially-robustnetwork 302. As can be seen, the difference between networks 300 and 302is the inclusion of an additional layer 316 that, as compared to theother intermediate layers, has reduced dimensionality. The notion ofdimensionality here is being used in its usual way, i.e., a reference tothe number of input variables or features for a particular dataset (inthis case the additional layer). As depicted in FIG. 3 , the additionallayer 316 is positioned in association with the output layer 312, inthis case right before the last hidden layer L3. This positioning is nota limitation, however, as the additional layer 316 may also bepositioned between the last hidden layer L3 and the output layer 312.Generalizing, the additional layer 316 is positioned then at or near theoutput layer 312 of the neural network. As used herein, the notion ofhaving reduced dimensionality means that the additional layer has alesser number of neurons as compared to at least the last intermediatelayer to which it is coupled. Although the dimensionality of theadditional layer will vary (based on the size of the intermediatelayer(s) themselves, preferably the additional layer has significantlylower dimensionality than the one or more (or at least an adjacent)intermediate layer. As will be described, preferably the additionallayer 316 is a fully-connected layer. Further, although only oneadditional layer 316 is depicted in FIG. 3 , there may be one or moreadditional layers that are similar. In one non-limiting embodiment, afully-connected layer 316 is added before a logit layer (e.g., L3) (orequivalently, after the penultimate layer). In this embodiment, thefully-connected layer has a sigmoid activation, although otheractivations (e.g., tanh) may be used.

As depicted, the additional layer 316 is sometimes referred to herein asan “n-hot” layer, as it implements an n-hot encoding scheme, as will befurther described, and that is an alternative to traditional one-hotencoding for classes. The number of neurons in the n-hot encoding layer316 preferably is less than the number of neurons in the previous layer,but also the number of neurons is not smaller than a minimum number ofbits required to express the number of classifier output classes. Forexample, if the classifier has ten (10) output classes, then the n-hotlayer 316 should have at least four (4) neurons (≥√{square root over(10)}) in order to uniquely label a class. An optimal size for the n-hotlayer preferably is determined during training, e.g., byexperimentation, or some preconfigured sizing may be used subject to theabove-described constraints. After adding the n-hot encoding layer 316to the model and determining the size of the layer, the n-hot encoding,which preferably is a vector of 0s and 1s, is the applied to the weightsof the layer. This encoding may be manually-constructed,randomly-generated, or provided through other means deterministically,e.g., using domain knowledge, an ontology, or a knowledge graph.

FIG. 4 depicts a process flow of one approach to generating theencoding, in this example randomly. At step 400, the size of the layerand the number of output classes is identified. As step 402, a randomn-hot vector for each output class is then generated, where the numberof hot bits (i.e., the number of 1s) is greater than 0. At step 404,these random p-hot vectors are introduced into the network as fixedweights for connections between the added n-hot layer and the finaloutput layer. More formally, the output of the n-hot encoding layer(where the value n may be set as a hyperparameter) is represented asy=Wx, where x is the input to the layer and W is the weight matrix, andthe rows of W are {E₁, E₂, . . . E_(n)}, where E_(i) is the i^(th)random n-hot encoding vector for output class i. In machine learning, ahyperparameter is a parameter whose value is used to control thelearning process. As noted, this encoding is sometimes referred toherein as n-hot encoding to distinguish it from traditional 1-hotencoding, and wherein the value of “n” preferably is configured ordesignated a hyperparameter. The neural network as augmented to includethis additional layer is then trained. This is step 406. Based on thetraining, the encoding causes the network to associate a reduced numberof active features with each of the hot positions (namely, the is in theweight matrix), thereby encouraging the network to learn those featuresthat contain a high amount of information with respect to each class.Thus, the additional layer limits (constrains) the network and, inparticular, on the number of features used to predict each class, and/orthe layer adds constraints between those features and the outputclasses. Once trained in this manner, the resulting adversarially-robustneural network is then output at step 405. At step 410, i.e. aftertraining, the adversarially-robust neural network is applied to aclassification task. As noted above, the nature of the classificationtask performed by the adversarially-robust network classifier varies,and typically it is dependent on the particular deployment or deploymentobjective.

Thus, according to this disclosure, one or more layers are added to theneural network and provide an alternative encoding to the traditionalone-hot encoding for classes. As described, the n-hot encoding appliedto an additional layer is used during training and encourages thenetwork to associate features (a feature set) with each of the hotpositions. In the example in FIG. 4 , wherein the encodings arerandomly-generated, random n-hot vectors are generated for each outputclass. These random n-hot vectors are introduced into the network asfixed weights for connections between the added n-hot layer and thefinal output layer. As the network is trained (typically in a normalmanner), the fixed encoding weights preferably are left untouched. Theoutput of the n-hot encoding layer is then represented by y=Wx, asdescribed above.

As noted, the particular placement of the alternative encoding layer mayvary. Typically, it is placed before the penultimate layer so as tominimize noise. That said, technically the layer may be placed anywherebefore the last layer. Irrespective of the particular placement, the lowdimensionality layer works to find relevant features. In particular, byconstraining an intermediate layer to be much lower (in dimensionality)as compared to a surrounding layer, the model in effect needs to performa compression and decompression (i.e., reconstruction) step around thelow dimensional layer. Thus, to maximize the accuracy of thedecompression the features that are learned in the low dimensional layershould be the features that contain the most information relevant toeach of the output classes. In the n-hot encoding framework asdescribed, it is possible to use either a pre-defined encoding or even arandom encoding (by default). When the encoding is then fixed, and givenan input for a certain class, the network has to identify what featuresin the layers prior to the n-hot layer can be combined to obtain thefixed encoding. This type of operation may be more readily apparent whena pre-defined encoding is used instead of random encodings (e.g.,digital clock encoding for digits), but the benefits are obtained inboth scenarios. In particular, and through training, the network learnsto extract the features defined in the n-hot layer rather than justrandomly learning a set of relevant features, which is the normaltraining method.

The technique described above has significant advantages. It is muchmore computationally-efficient as compared to prior data augmentationtechniques, and the approach does not rely on using non-differentiableconstructions to hide a loss gradient from an adversary. The techniqueis simple to implement, as the additional layer has a significantlylower dimensionality than other network layers, and the encoding schemeensures that the network learns the reduced number of active featuresquickly and reliably. Further, the approach does not require any changesto the existing training.

Another instantiation of the above-described technique is solving theproblem as multi-task learning that trains a classifier to classify theinput using multiple sets of classes. An example solution of multi-tasklearning involves training the above-described model with the weightedsum of the multiple loss functions regarding those sets of classes,including the original set of classes and auxiliary/feature sets ofclasses. This approach enhances the learned encoding of the input asrobust and semantically meaningful, and it also forces the model to usegeneralizable encoding. For example, consider an image classifierclassifying a given image into one of {bird, airplane, dog}. Then,instead of training the classifier to just classify the input into oneof the three classes, the approach also may consider other sets, such as{wings, no wings}, and {animal, plant, inanimate}.

The above-described variant embodiment introduce constraints on theshared encoding by the loss functions while each loss function iscomputed independently. Instead of using each loss functionindependently, another approach is to first train a neural network toclassify the input to feature classes using multi-task learning, andthen building a hyper-classifier on top of the inputs with the originalset of classes. Then, the hyper-classifier is constrained to use onlythe extracted features; the hyper-classifier can then be trained or havemanual mappings assigned to it. More generally, this approach involvestraining a classifier to map inputs to features, and then train a newclassifier to map the features to output classes. This provides anend-to-end classification pipeline. In any of these variant embodiments,the auxiliary/feature sets of classes or the mappings from such featuresets to the original target classes can be learned using training data,manually crafted, or extracted from an ontology or knowledge graph.

The technique herein may be implemented as an architecture modification,alone or in combination with other existing adversarial defenses such asdata augmentation (adversarial training, Gaussian smoothing, andothers).

One or more aspects of this disclosure (e.g., augmenting the NN, thetesting for adversarial samples, etc.) may be implemented as-a-service,e.g., by a third party. The subject matter may be implemented within orin association with a data center that provides cloud-based computing,data storage or related services.

In a typical use case, a SIEM or other security system has associatedtherewith an interface that can be used to issue API queries to thetrained model, and to receive responses to those queries includingresponses indicator of adversarial input.

The approach herein is designed to be implemented on-demand, or in anautomated manner.

Access to the service for model training or use to identify adversarialinput may be carried out via any suitable request-response protocol orworkflow, with or without an API.

FIG. 5 depicts an exemplary distributed data processing system in whichthe deployed system or any other computing task associated with thetechniques herein may be implemented. Data processing system 500 is anexample of a computer in which computer-usable program code orinstructions implementing the processes may be located for theillustrative embodiments. In this illustrative example, data processingsystem 500 includes communications fabric 502, which providescommunications between processor unit 504, memory 506, persistentstorage 505, communications unit 510, input/output (I/O) unit 512, anddisplay 514.

Processor unit 504 serves to execute instructions for software that maybe loaded into memory 506. Processor unit 504 may be a set of one ormore processors or may be a multi-processor core, depending on theparticular implementation. Further, processor unit 504 may beimplemented using one or more heterogeneous processor systems in which amain processor is present with secondary processors on a single chip. Asanother illustrative example, processor unit 504 may be a symmetricmulti-processor (SMP) system containing multiple processors of the sametype.

Memory 506 and persistent storage 505 are examples of storage devices. Astorage device is any piece of hardware that is capable of storinginformation either on a temporary basis and/or a permanent basis. Memory506, in these examples, may be, for example, a random access memory orany other suitable volatile or non-volatile storage device. Persistentstorage 505 may take various forms depending on the particularimplementation. For example, persistent storage 505 may contain one ormore components or devices. For example, persistent storage 505 may be ahard drive, a flash memory, a rewritable optical disk, a rewritablemagnetic tape, or some combination of the above. The media used bypersistent storage 505 also may be removable. For example, a removablehard drive may be used for persistent storage 505.

Communications unit 510, in these examples, provides for communicationswith other data processing systems or devices. In these examples,communications unit 510 is a network interface card. Communications unit510 may provide communications through the use of either or bothphysical and wireless communications links.

Input/output unit 512 allows for input and output of data with otherdevices that may be connected to data processing system 500. Forexample, input/output unit 512 may provide a connection for user inputthrough a keyboard and mouse. Further, input/output unit 512 may sendoutput to a printer. Display 514 provides a mechanism to displayinformation to a user.

Instructions for the operating system and applications or programs arelocated on persistent storage 505. These instructions may be loaded intomemory 506 for execution by processor unit 504. The processes of thedifferent embodiments may be performed by processor unit 504 usingcomputer implemented instructions, which may be located in a memory,such as memory 506. These instructions are referred to as program code,computer-usable program code, or computer-readable program code that maybe read and executed by a processor in processor unit 504. The programcode in the different embodiments may be embodied on different physicalor tangible computer-readable media, such as memory 506 or persistentstorage 505.

Program code 516 is located in a functional form on computer-readablemedia 515 that is selectively removable and may be loaded onto ortransferred to data processing system 500 for execution by processorunit 504. Program code 516 and computer-readable media 515 form computerprogram product 520 in these examples. In one example, computer-readablemedia 515 may be in a tangible form, such as, for example, an optical ormagnetic disc that is inserted or placed into a drive or other devicethat is part of persistent storage 505 for transfer onto a storagedevice, such as a hard drive that is part of persistent storage 505. Ina tangible form, computer-readable media 515 also may take the form of apersistent storage, such as a hard drive, a thumb drive, or a flashmemory that is connected to data processing system 500. The tangibleform of computer-readable media 515 is also referred to ascomputer-recordable storage media. In some instances,computer-recordable media 515 may not be removable.

Alternatively, program code 516 may be transferred to data processingsystem 500 from computer-readable media 515 through a communicationslink to communications unit 510 and/or through a connection toinput/output unit 512. The communications link and/or the connection maybe physical or wireless in the illustrative examples. Thecomputer-readable media also may take the form of non-tangible media,such as communications links or wireless transmissions containing theprogram code. The different components illustrated for data processingsystem 500 are not meant to provide architectural limitations to themanner in which different embodiments may be implemented. The differentillustrative embodiments may be implemented in a data processing systemincluding components in addition to or in place of those illustrated fordata processing system 500. Other components shown in FIG. 5 can bevaried from the illustrative examples shown. As one example, a storagedevice in data processing system 500 is any hardware apparatus that maystore data. Memory 506, persistent storage 505, and computer-readablemedia 515 are examples of storage devices in a tangible form.

In another example, a bus system may be used to implement communicationsfabric 502 and may be comprised of one or more buses, such as a systembus or an input/output bus. Of course, the bus system may be implementedusing any suitable type of architecture that provides for a transfer ofdata between different components or devices attached to the bus system.Additionally, a communications unit may include one or more devices usedto transmit and receive data, such as a modem or a network adapter.Further, a memory may be, for example, memory 506 or a cache such asfound in an interface and memory controller hub that may be present incommunications fabric 502.

The techniques herein may be used with a host machine (or set ofmachines, e.g., running a cluster) operating in a standalone manner, orin a networking environment such as a cloud computing environment. Cloudcomputing is an information technology (IT) delivery model by whichshared resources, software and information are provided over theInternet to computers and other devices on-demand. With this approach,an application instance is hosted and made available from Internet-basedresources that are accessible through a conventional Web browser ormobile application over HTTP. Cloud compute resources are typicallyhoused in large server farms that run one or more network applications,typically using a virtualized architecture wherein applications runinside virtual servers, or so-called “virtual machines” (VMs), that aremapped onto physical servers in a data center facility. The virtualmachines typically run on top of a hypervisor, which is a controlprogram that allocates physical resources to the virtual machines.

Typical cloud computing service models are as follows:

Software as a Service (SaaS): the capability provided to the consumer isto use the provider's applications running on a cloud infrastructure.The applications are accessible from various client devices through athin client interface such as a web browser (e.g., web-based e-mail).The consumer does not manage or control the underlying cloudinfrastructure including network, servers, operating systems, storage,or even individual application capabilities, with the possible exceptionof limited user-specific application configuration settings.

Platform as a Service (PaaS): the capability provided to the consumer isto deploy onto the cloud infrastructure consumer-created or acquiredapplications created using programming languages and tools supported bythe provider. The consumer does not manage or control the underlyingcloud infrastructure including networks, servers, operating systems, orstorage, but has control over the deployed applications and possiblyapplication hosting environment configurations.

Infrastructure as a Service (IaaS): the capability provided to theconsumer is to provision processing, storage, networks, and otherfundamental computing resources where the consumer is able to deploy andrun arbitrary software, which can include operating systems andapplications. The consumer does not manage or control the underlyingcloud infrastructure but has control over operating systems, storage,deployed applications, and possibly limited control of select networkingcomponents (e.g., host firewalls).

Typical deployment models are as follows:

Private cloud: the cloud infrastructure is operated solely for anorganization. It may be managed by the organization or a third party andmay exist on-premises or off-premises.

Community cloud: the cloud infrastructure is shared by severalorganizations and supports a specific community that has shared concerns(e.g., mission, security requirements, policy, and complianceconsiderations). It may be managed by the organizations or a third partyand may exist on-premises or off-premises.

Public cloud: the cloud infrastructure is made available to the generalpublic or a large industry group and is owned by an organization sellingcloud services.

Hybrid cloud: the cloud infrastructure is a composition of two or moreclouds (private, community, or public) that remain unique entities butare bound together by standardized or proprietary technology thatenables data and application portability (e.g., cloud bursting forload-balancing between clouds).

Some clouds are based upon non-traditional IP networks. Thus, forexample, a cloud may be based upon two-tier CLOS-based networks withspecial single layer IP routing using hashes of MAC addresses. Thetechniques described herein may be used in such non-traditional clouds.

The system, and in particular the modeling and consistency checkingcomponents, typically are each implemented as software, i.e., as a setof computer program instructions executed in one or more hardwareprocessors. The components may also be integrated with one another inwhole or in part. One or more of the components may execute in adedicated location, or remote from one another. One or more of thecomponents may have sub-components that execute together to provide thefunctionality. There is no requirement that particular functions beexecuted by a particular component as named above, as the functionalityherein (or any aspect thereof) may be implemented in other or systems.

The approach may be implemented by any service provider that operatesinfrastructure. It may be available as a managed service, e.g., providedby a cloud service. A representative deep learning architecture of thistype is IBM® Watson® Studio.

The components may implement the workflow synchronously orasynchronously, continuously and/or periodically.

The approach may be integrated with other enterprise- or network-basedsecurity methods and systems, such as in a SIEM, APT, graph-basedcybersecurity analytics, or the like.

Computer program code for carrying out operations of the presentinvention may be written in any combination of one or more programminglanguages, including an object-oriented programming language such asJava™, Smalltalk, C++ or the like, and conventional proceduralprogramming languages, such as the “C” programming language or similarprogramming languages. The program code may execute entirely on theuser's computer, partly on the user's computer, as a stand-alonesoftware package, partly on the user's computer and partly on a remotecomputer, or entirely on the remote computer or server. In the latterscenario, the remote computer may be connected to the user's computerthrough any type of network, including a local area network (LAN) or awide area network (WAN), or the connection may be made to an externalcomputer (for example, through the Internet using an Internet ServiceProvider).

Those of ordinary skill in the art will appreciate that the hardware inFIG. 5 may vary depending on the implementation. Other internal hardwareor peripheral devices, such as flash memory, equivalent non-volatilememory, or optical disk drives and the like, may be used in addition toor in place of the hardware depicted. Also, the processes of theillustrative embodiments may be applied to a multiprocessor dataprocessing system, other than the SMP system mentioned previously,without departing from the spirit and scope of the disclosed subjectmatter.

The functionality described in this disclosure may be implemented inwhole or in part as a standalone approach, e.g., a software-basedfunction executed by a hardware processor, or it may be available as amanaged service (including as a web service via a SOAP/XML interface).The particular hardware and software implementation details describedherein are merely for illustrative purposes are not meant to limit thescope of the described subject matter.

More generally, computing devices within the context of the disclosedsubject matter are each a data processing system (such as shown in FIG.5 ) comprising hardware and software, and these entities communicatewith one another over a network, such as the Internet, an intranet, anextranet, a private network, or any other communications medium or link.

The scheme described herein may be implemented in or in conjunction withvarious server-side architectures including simple n-tier architectures,web portals, federated systems, and the like. The techniques herein maybe practiced in a loosely-coupled server (including a “cloud”-based)environment.

Still more generally, the subject matter described herein can take theform of an entirely hardware embodiment, an entirely software embodimentor an embodiment containing both hardware and software elements. In apreferred embodiment, the function is implemented in software, whichincludes but is not limited to firmware, resident software, microcode,and the like. Furthermore, as noted above, the identity context-basedaccess control functionality can take the form of a computer programproduct accessible from a computer-usable or computer-readable mediumproviding program code for use by or in connection with a computer orany instruction execution system. For the purposes of this description,a computer-usable or computer readable medium can be any apparatus thatcan contain or store the program for use by or in connection with theinstruction execution system, apparatus, or device. The medium can be anelectronic, magnetic, optical, electromagnetic, infrared, or asemiconductor system (or apparatus or device). Examples of acomputer-readable medium include a semiconductor or solid state memory,magnetic tape, a removable computer diskette, a random access memory(RAM), a read-only memory (ROM), a rigid magnetic disk and an opticaldisk. Current examples of optical disks include compact disk-read onlymemory (CD-ROM), compact disk-read/write (CD-R/W) and DVD. Thecomputer-readable medium is a tangible item.

In a representative embodiment, the techniques described herein areimplemented in a special purpose computer, preferably in softwareexecuted by one or more processors. The software is maintained in one ormore data stores or memories associated with the one or more processors,and the software may be implemented as one or more computer programs.Collectively, this special-purpose hardware and software comprises thefunctionality described above.

While the above describes a particular order of operations performed bycertain embodiments, it should be understood that such order isexemplary, as alternative embodiments may perform the operations in adifferent order, combine certain operations, overlap certain operations,or the like. References in the specification to a given embodimentindicate that the embodiment described may include a particular feature,structure, or characteristic, but every embodiment may not necessarilyinclude the particular feature, structure, or characteristic.

Finally, while given components of the system have been describedseparately, one of ordinary skill will appreciate that some of thefunctions may be combined or shared in given instructions, programsequences, code portions, execution threads, and the like.

The techniques herein provide for improvements to another technology ortechnical field, e.g., deep learning systems, real-world applications ofdeep learning models including, without limitation, medicalclassifications, other security systems, as well as improvements todeployed systems that use deep learning models to facilitate command andcontrol operations with respect to those deployed systems.

As previously mentioned, the technique herein may be used in any domainand with any application wherein the neural network classifier may besubject to adversarial attack.

The techniques described herein are not limited for use with anyparticular type of deep learning model. The approach may be extended toany machine learning model including, without limitation, a SupportVector Machine (SVM), a logistical regression (LR) model, and the like,that has internal processing states (namely, hidden weights), and theapproach may also be extended to use with decision tree-based models.

The particular classification task that may be implemented is notintended to be limited. Representative classification tasks include,without limitation, image classification, text recognition, speechrecognition, natural language processing, and many others.

Having described the subject matter, what we claim is as follows.

The invention claimed is:
 1. A method to constrain and operate a neuralnetwork to enhance robustness against adversarial attack, the neuralnetwork comprising a first layer, a last layer, and one or moreintermediate layers, comprising: augmenting the neural network byassociating a fully-connected additional layer with the last layer,wherein the additional layer has a lower dimensionality than at leastone intermediate layer; applying an encoding to the additional layer,wherein the encoding comprises an encoding vector for each output class;training the neural network having the additional layer and the appliedencoding to learn a reduced feature set representing one or morefeatures containing information with respect to at least one outputclass; deploying the trained neural network in association with adecision-making computer system or application; and performing aclassification using the trained neural network, thereby supportingdecision-making by the decision-making computer system or application.2. The method as described in claim 1 wherein the encoding is a vectorbit encoding scheme comprising a set of bit vectors, and wherein ani^(th) encoding vector of the set of bit vectors represents an i^(th)output class of the neural network.
 3. The method as described in claim2 further including: associating a set of fixed encoding weights forconnections between the additional layer and the output layer; andmaintaining the fixed encoding weights unchanged by the training.
 4. Themethod as described in claim 1 wherein the training also adds one ormore constraints between the one or more features and the output layer.5. The method as described in claim 1 wherein the additional layer isfully connected using a sigmoid activation and is positioned between alast intermediate layer and the last layer, wherein the last layer is alogit function layer.
 6. The method as described in claim 1 wherein thetraining associates the one or more features for the at least outputclass with a particular bit position in the encoding vector.
 7. Themethod as described in claim 1 further including sizing the additionallayer to include a number of neurons sufficient to encode a unique labelfor each output class of the neural network.
 8. The method as describedin claim 1 wherein the training uses a weighted sum of multiple lossfunctions for a set of classes.
 9. An apparatus, comprising: aprocessor; computer memory holding computer program instructionsexecuted by the processor to constrain and operate a neural network toenhance robustness against adversarial attack, the computer programinstructions configured to: associate a fully-connected additional layerwith the last layer to augment the neural network, wherein theadditional layer has a lower dimensionality than at least oneintermediate layer; apply an encoding to the additional layer, whereinthe encoding comprises an encoding vector for each output class; trainthe neural network having the additional layer and the applied encodingto learn a reduced feature set representing one or more features thatcontain information with respect to at least one output class; deploythe trained neural network in association with a decision-makingcomputer system or application; and perform a classification using thetrained neural network to support decision-making by the decision-makingcomputer system or application.
 10. The apparatus as described in claim9 wherein the encoding is a vector bit encoding scheme comprising a setof bit vectors, and wherein an i^(th) encoding vector of the set of bitvectors represents an i^(th) output class of the neural network.
 11. Theapparatus as described in claim 9 wherein the computer programinstructions are further configured: associate a set of fixed encodingweights for connections between the additional layer and the outputlayer; wherein the fixed encoding weights remain unchanged by thetraining.
 12. The apparatus as described in claim 9 wherein the trainingalso adds one or more constraints between the one or more features andthe output layer.
 13. The apparatus as described in claim 9 wherein theadditional layer is fully connected using a sigmoid activation and ispositioned between a last intermediate layer and the last layer, whereinthe last layer is a logit function layer.
 14. The apparatus as describedin claim 9 wherein the training associates the one or more features forthe at least output class with a particular bit position in the encodingvector.
 15. The apparatus as described in claim 9 wherein the computerprogram instructions are further configured to size the additional layerto include a number of neurons sufficient to encode a unique label foreach output class of the neural network.
 16. The apparatus as describedin claim 9 wherein the training uses a weighted sum of multiple lossfunctions for a set of classes.
 17. A computer program product in anon-transitory computer readable medium for use in a data processingsystem to constrain and operate a neural network to enhance robustnessagainst adversarial attack, the neural network comprising a first layer,a last layer, and one or more intermediate layers, the computer programproduct holding computer program instructions that, when executed by thedata processing system, are configured to: associate a fully-connectedadditional layer with the last layer to augment the neural network,wherein the additional layer has a lower dimensionality than at leastone intermediate layer; apply an encoding to the additional layer,wherein the encoding comprises an encoding vector for each output class;train the neural network having the additional layer and the appliedencoding to learn a reduced feature set representing one or morefeatures that contain information with respect to at least one outputclass; deploy the trained neural network in association with adecision-making computer system or application; and perform aclassification using the trained neural network to supportdecision-making by the decision-making computer system or application.18. The computer program product as described in claim 17 wherein theencoding is a vector bit encoding scheme comprising a set of bitvectors, and wherein an i^(th) encoding vector of the set of bit vectorsrepresents an i^(th) output class of the neural network.
 19. Thecomputer program product as described in claim 17 wherein the computerprogram instructions are further configured: associate a set of fixedencoding weights for connections between the additional layer and theoutput layer; wherein the fixed encoding weights remain unchanged by thetraining.
 20. The computer program product as described in claim 17wherein the training also adds one or more constraints between the oneor more features and the output layer.
 21. The computer program productas described in claim 17 wherein the additional layer is fully connectedusing a sigmoid activation and is positioned between a last intermediatelayer and the last layer, wherein the last layer is a logit functionlayer.
 22. The computer program product as described in claim 17 whereinthe training associates the one or more features for the at least outputclass with a particular bit position in the encoding vector.
 23. Thecomputer program product as described in claim 17 wherein the computerprogram instructions are further configured to size the additional layerto include a number of neurons sufficient to encode a unique label foreach output class of the neural network.
 24. The computer programproduct as described in claim 17 wherein the training uses a weightedsum of multiple loss functions for a set of classes.